Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models

Many modern Language Model (LM) pipelines return an averaged model, such as an exponential moving average of the training iterates, rather than the final iterate itself. This raises a fundamental question: given that we will return an iterate average, how should we change training to improve the per…

View Post on Scour ↗

More from arXiv on Scour.